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MULTI-PURPOSE SWITCHING 
NETWORK INTERFACE CONTROLLER 

TECHNICAL FIELD 
The present invention is in the field of network interface controllers and, in 
5 particular, is related to a switching network interface controller that is configurable for use 
in a variety of different networking configurations. 

BACKGROUND 

There are a variety of networking applications for which, conventionally, various 
specialized hardware is available. It would be desirable to have a network interface 
10 controller solution that is flexibly configured for use in these various applications. 

SUMMARY 

A reconfigurable network interface controller circuit is provided for various 
networking operations. The circuit is scalable, and provides for redundancy in such 
networking operations. 

15 The network interface controller includes a plurality of scatter gather circuits 

connectable to a host via a bus. A packet buffer is configured for communication with the 
scatter gather circuits. A plurality of access circuits are configured to access external 
network connections. An optional forwarding engine is selectable to generate routing 
information corresponding to data received via the access circuits and to provide the 

20 routing information to the packet buffer. 

BRIEF DESCRIPTION OF FIGURES 
Figure 1 schematically illustrates an example multipurpose network interface 
controller circuit. 

Figures 2A and 2B illustrate the Figure 1 circuit in a network interface controller 
25 configuration. 

Figures 3A and 3B illustrate the Figure 1 circuit in a switch configuration. 

Figure 4 illustrate the Figure 1 circuit in a hybrid configuration. 

Figure 5 illustrates the Figure 1 circuit in a network processor configuration. 
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DETAILED DESCRIPTION 
Turning now to Figure 1, an example configurable network interface controller 
circuit 100 is described. In some embodiments, the Figure 1 network interface controller 
circuit is implemented in an application specific integrated circuit (ASIC), such as a field- 
programmable gate array (FPGA). In one example, the circuit is configurable to implement 
applications such as: 

• Multi-Channel Ethernet NIC Controller 

• L2/L3 Ethernet Switch 

• System Controller 

• SOHO Router 

• Network Processor 

The various blocks of the Figure 1 example are now summarized. 

PCI-XCORE(PX102) 

This is a standard quad-function PCI-X core, with 4 identical Master interfaces to 

15 SGE's 1 04a through 1 04d, and one target interface TI 1 06. This block in addition houses 

the Serial ROM interface logic. 

Forwarding Engine (FE 1 08) 

This module makes routing decisions based on a vector that it receives from the 

MAC 1 10 or the SGE 104 modules. The results of the routing decisions are conveyed to the 

20 packet buffer PB 1 12 via the Route Vector bus 1 14. 

MAC (Mn 110) 

This is a 10/100/1000 MAC module. In one example, this module includes VLAN 
Tag and Station Address logic (which are normally disabled in the presence of a 
Forwarding Engine). 

25 PacketBuffer(PB112) 

This is a general purpose queuing engine core switch fabric, that allows 

management of arbitrary sized packets. A particular example of the PB 1 12 is described in 

U.S. Provisional Patent Application No. 60/283,285 filed April 11, 2001 and incorporated 

herein by reference in its entirety. 

2 
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Memory Controller (MC 116) 

This is a generic memory controller that can control, for example, SSRAM 

memories 118. 

Lookup Memory Controller (LM 120) 
5 This is an SSRAM only memory controller that provides a path to an external 

memory 122 (e.g., for a forwarding table and/or instruction memory). 

CPU Interface (CI 124) 

This module is a collection of muxes and base addresses that switches various 

busses and manages the address map of an on-board processor 125. This module decodes 

10 the PL-Bus regions and generates chip selects to various modules. 

Scatter Gather Engine (SGn 104) 

This module handles the Scatter Gather operation in a NIC mode, provides a path to 

the packet buffer 1 12 from PX 102 in the Switch mode, and provides a path to the packet 

buffer 1 12 from the CPU 125 in the System Controller mode. 

15 Wake up, On Now Logic (ON 124) 

This module implements power management functions. 

Narrow Checksum (NC 126) 

This module calculates the TCP/IP/HTTP/UDP checksums and provides the result 
to the packet buffer 112 upon completion of a frame arriving from the MAC 1 10. 

20 Wide Checksum (WC) 

This module calculates the TCP/IP/HTTP/UDP checksums and provides the result 
to the packet buffer PB 1 12 upon the completion of a frame as it is arriving from the PCI 
bus 101 via the PX 102. 

25 In all modes of the operation (NIC, Switch, etc.), the data is pushed to the SGE 1 04 

by the packet buffer 1 12 and to the packet buffer 1 12 by the SGE 104. 

Clocking within the Figure 1 example is now described. Specifically, the 
components of the Figure 1 example operate in three different clock domains. These clock 
domains include the core clock domain (which includes the on-board CPU 125 and the 

30 memory interfaces (LM 120 and MEMC 116); the MAC 110 clock domains (which, in 

3 
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some examples, includes several actual clock regimes); and the PCI-X 101 clock domain. 
In some examples, no phase lock loops are employed. 

In cases where the Figure 1 example is an FPGA as opposed to an ASIC, the FPGA 
clocking is the same as the ASIC clocking except for the MAC clock section. Further, 
5 some 10 delay lines are inherent to the FPGA 10 pads, and are explicitly built in the ASIC 
configuration. 

In reset, all clock domains are synchronized to the PCI-X clock domain. The PHY 
chips can be reset individually. In addition, soft reset resources allow the local resetting of 
the Figure 1 circuitry by the local processor 125 under control of local firmware. 

10 In accordance with one example, the Figure 1 circuit is operable in at least four 

different modes by reconfiguring the operation of the various modules. Each of these four 
modes of operation are now described. In the description, the Receive and Transmit 
nomenclatures are always with reference to the wire (i.e., receive from the wire and 
transmit to the wire) unless otherwise specified. 

1 5 The NIC operation mode is now discussed with reference to Figures 2 A and 2B. In 

general, the NIC operation mode is a relatively simple mode of operation of the Figure 1 
example circuit. A simplified block diagram of the NIC mode is represented in Figures 2A 
and 2B. In this mode, the on-chip processor 124, the forwarding engine 108 and the 
associated secondary memory 122 are not used (and, in fact, need not be even populated in 

20 the ASIC/FPGA) and, thus, are shown as "grayed out" in Figure 2B. Various NIC 
operation mode operations are now described. 

NIC Receive 

In the NIC receive operation, data is received from the MAC 110, through the N- 
. Bus, and stored in the PB 1 12 using, for example, the paging hardware of the packet buffer 

25 PB 1 12. The Rvec is determined in one example using a stub model, for the packet buffer 
PB 1 12 to determine from which queue it should push the data out. Once the data is 
completely stored in the appropriate queues of the packet buffer, it is then pushed to the 
appropriate Scatter Gather Engine 1 04. While transiting on the N-Bus, checksums are 
calculated on the incoming data from the MAC 1 1 0 by the NCS 126 and stored into the 

30 packet buffer PB 1 12. In one example, the checksum 1 1 0 storage is accomplished by 

strobing a payload (including the checksum) to the packet buffer PB 1 12 following the last 
element of the data being stored. The stored data is then transferred into the host memory 

4 
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from the PB. 1 12 by the Scatter Gather Engine 104 (via the PX 102 and the PCI-X 101). In 
this mode of operation, the packet buffer PB 1 12 does not perform any broadcast or 
multicast operations and, as such, looks substantially like an external FIFO per channel. 

NIC Transmit 

5 In the NIC transmit mode, data is gathered from the host memory (via the PCI-X 

101 and PX 102) by the appropriate SGE 104 and pushed to the packet buifer PB 112. 
While transiting the W-Bus, checksums are calculated on the data by the WCS 128 and 
delivered to the packet buffer PB 1 12 by strobing a payload (including the checksum) into 
the packet buffer PB 1 12 following storage of the last element of data. Once the packet is 

10 completely stored in the packet buffer PB 1 12, the packet is forwarded onto the N-Bus to 
the appropriate MAC 110. 

NIC Fail Over 

The link fail over is handled entirely in software. 

NIC Multicast and Broadcast 
15 These features are handled entirely in software. 

NIC Flow Control 

In the receive direction, once the descriptor queue of a TAP in the packet buffer PB 
1 12 goes over a high water mark, a signal is asserted to the corresponding MAC 1 10 to 
cause the MAC 1 1 0 to generate flow control signals to its link partner. Depending on the 

20 MAC configuration, the traffic resumes either after a fixed amount of time or after the 
, queue in the packet buffer PB 1 12 has dropped below the low water mark, the occurrence 
of which is also transmitted to the.MAC 1 10. In the transmit direction, once the queue for 
the TAP, in the packet buffer, becomes full and data "backs up" into an SGE 104, the 
transmit function of the packet buffer 112 simply stops. In both directions, once the SGE 

25 1 04 runs out of resources such as free lists or transmit buffers, traffic simply stops. 

The switch operation mode is now described with reference to Figures 3A and 3B. 
In the switch configuration, the Figure 1 example circuit employs the local forwarding 
engine FE 108 to make routing decisions for each packet in each direction. A given packet 
30 is examined once upon arrival to the source node and once upon exiting the destination 
node. The local processor CPU 125 is used for management purposes. 

5 
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The switch backplane may be thought of as a virtual backplane. That is, the switch 
backplane includes multiple virtual channels on top of the PCI bus 101. Each virtual 
channel includes a scatter gather engine SGE 1 04 performing DMA writes to a reserved 
range of addresses in the host memory. That range of address is programmed into the 
5 receiving node's PCI base address register. Therefore, each pairing of the base address 
value programmed into a node's PCI base address register and the value programmed into 
another node's scatter gather engine 104 receive DMA address register configures a virtual 
channel. Each node is effectively a quad-function PCI device and hence has 4 memory 
base addresses, thus allowing 4 virtual full-duplex links to a node. This virtual back-plane 

10 of one example uses three base addresses, leaving the fourth one, corresponding to the node 
number of the device, unused. 

In the switch mode of operation, the Scatter Gather Engine 104 does not do any 
linked list operations and simply pushes the data as consecutive data elements to the PCI 
101. The data arriving from the PCI 101 is received on the PCI's target interface 106 which 

15 is then muxed to the appropriate SGE 1 04. The receive DMA address restarts at the base 
address at the start of every packet and the lower bits of the address are used to determine 
the length of a packet. Upper bits of the address, while still within the address range, are 
used to transfer additional information to the receiver. Each base address range is 256 
Mbytes (28 bits). The largest packet that the switch handles is 64Kbytes (16 bits), allowing 

20 12 bits of payload (e.g., sequence numbers, etc.). Each node can be conceptualized, in 
some instances, as either a Source Node or a Destination Node. Source Node is the node 
where a packet initially comes into the system (via the Source Port), and Destination Node 
is the node through which the packet leaves the switch. 

As data arrives from the MAC 1 10, the MAC 110 generates a long vector 

25 comprised of the DA, SA, VLAN, QoS fields of the incoming packet. This vector is then 
submitted to the forwarding engine FE 108. The packet then bypasses the checksum 126 
and wake up 124 blocks before being written into the packet buffer PB 112. By this time, 
the forwarding engine 1 08 computes a routing vector 114 which is provided to the packet 
buffer 1 12 for the packet buffer i 12 to place on the appropriate destination queue. In one 

30 example, a limit of 1 6 clocks is placed on the forwarding engine 1 08 for the lookup time 
per minimum sized packet (64 bytes). During this time, the forwarding engine 108 
completes two lookups and a conditional learn operation. 
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Once the packet is fully stored in the packet buffer 1 12, it is pushed out one of the 
W-Buses to the SGE module 104. The SGE module 104 — which operates as a pass 
through in the switch configuration - simply pushes the data out to the node that was 
indicated by the forwarding engine 108. The packet eventually appears on the target 
5 interface (T-Bus) of the destination node's PCI bus 101. Once the data arrives from the PCI 
bus 101 on the T-Bus of a destination node, the target interface 106 of the destination node 
routes the packet to the packet buffer 1 12 of the destination node. Meanwhile, the 
forwarding engine 108 computes a routing vector 1 14 for the packet and submits the 
routing vector 1 14 to the packet buffer 112. 

10 Switch multicast and broadcast are handled in the forwarding engine 108. In case 

of a multicast or broadcast packet, the result is flagged to the packet buffer via a routing 
vector with multiple bits set. 

In one example, the packet buffer 1 12 and the forwarding engine 108 can each 
pipeline two messages, to absorb the latency of the forwarding engine 108. The SGE 104 

15 use flow control PIO writes to its link partner SGE to control the rate of the switch 
backplane traffic. 

The hybrid mode of operation is now described with reference to Figure 4. In the 
hybrid mode of operation, the Figure 1 exampie circuit acts as a 4-port switch combined 
with a up to a 4-port NIC. The Figure 1 example circuit can "own" up to eight MAC 

20 addresses, four for its switched ports and up to four for its embedded NIC ports. The hybrid 
mode of operation is handled in the forwarding engine 1 08. Although the architecture 
allows for more, in some examples, the forwarding engine 108 has only enough bandwidth 
for a single port NIC. Power management logic is used in the hybrid mode to detect 
"magic packets," etc. and to generate appropriate power management events. 

25 Once a packet arrives from the MAC 110 ("receive event"), the received packet is 

submitted to the forwarding engine 108. The received packet is concurrently written into 
the packet buffer 112 followed by checksum information. The Rvec 1 14 from the 
forwarding engine 108 indicates if the packet should be switched out of a port (via one of 
the N-Buses) or delivered to the host (via a W-Bus). Data received into the SGE 1 04 from 

30 the packet buffer 1 1 2 is typically delivered to the host. 

For transmit, the data to be transmitted is normally gathered from the host memory 

into the packet buffer 1 12. However, the SGE 104 also submits the packet to the 

forwarding engine 108. The forwarding engine 108 in turn indicates to the packet buffer 

7 
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112 onto which queue of the packet buffer 1 12 to place the packet. The checksums are 
calculated in this path.. Although unlikely, one NIC port may wish to send a packet to 
another NIC port. 

In the "system controller" mode of operation, any external host adapter is leveraged 
5 for interfacing to other protocols (e.g., SCSI, USB, SONET, USB, etc.). The NIC appear as 
a PC to the external host adapters. As such, the Figure 1 example circuit provides a high 
bandwidth path to the local packet buffer 1 12. Further, a section of the packet buffer 1 12 is 
allocated as simple passive memory for use by the external host adapters. The external host 
adapters use this area to scatter and gather their traffic into the local memory 118. The 

10 processor 125 can access this data via the C-Bus path. Thus, in this mode, the Figure 1 
circuit is functioning as a system controller integrated with a processor. 

In the network processor mode of operation, represented in Figure 5, firmware 
effectively "visits" every packet at the wire speed while relieving some of the overhead of 
packet processing. All the packet processing may then be performed via the firmware. This 

15 mode of operation can be thought of as a superset of the system controller mode of 

operation. In this mode of operation, it is generally desired to transfer data from one MAC 
1 10 to another MAC 1 10 or to an external host adapter once the processor has had an 
opportunity to read and write portions of the packet. Once the packet is modified, the 
Figure 1 circuit recomputes the checksums. In this configuration, the SGE modules 104 are 

20 used by the local processor 125 to access the packets via the queueing system. Thus, the 
head of the W-Bus FIFO in the SGE module 104 is read by the processor 125 via C-Bus 
and it is written to via C-Bus. 

The network processor receive operation is now described. As packets arrive from 
the MAC 1 10, the checksums for the packets are computed (126) and the packets are 

25 written into the packet buffer 1 12. The packet buffer 1 12 operates in the NIC mode in this 
case. That is, there is effectively a hard path between N-Bus-0 to W-Bus-0, etc. The 
received data is pushed out to the respective SGE 104 via the respective W-Bus. Once the 
SGE 104 senses the presence of new data, it interrupts the local processor 125. The 
processor 125 then performs burst reads of SGE's 1 04 W-Bus receive FIFO into the data 

30 cache of the processor 125. Once the processor 125 has examined and rewritten the data, 

the processor 125 scatters the data into a portion of the packet buffer 1 12 not used for 

queuing. The data is then gathered by the external host adapter and consumed by the host 

adapter. Optionally, the data from the local processor 125 is burst written into the SGE's 

8 
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104 transmit W-Bus FIFO. The checksum is computed (128) on this data by the Figure 1 
example circuit before being written into the packet buffer 1 12. Further, the routing vector 
from the first element of data is retrieved and submitted to forwarding engine 108 for 
multiplexing onto the Rvec bus 1 14 so as to place the packet on the desired N-Bus or W- 
5 Bus queue in the PB 1 1 2. 

The network processor transmit operation is now described. When a packet is to be 
transmitted to the wire by an external host adapter, the packet is first scattered into the local 
memory 1 1 8 by the external host adapter. The packet is then read by the local processor 
125 via the C-Bus. Once the local processor 125 has examined and rewritten the packet, . 

10 the local processor 125 burst writes the packets along with a routing vector 1 14 into the 
transmit W-Bus queue of the SGE 104. The checksum on the packet is then computed 
(128) and the packet is forwarded to the N-Bus or W-Bus. If the packet to be transmitted is 
coming from another port of the same instantiation of the Figure 1 circuit, the packet is first 
received by the local processor 125 and is then burst written into the SGE 104 along with a 

15 routing vector. 
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What is claimed is: 

1 . A network interface controller, comprising: 

a plurality of scatter gather circuits connectable to a host via a bus; 

a packet buffer configured for communication with the scatter gather circuits; 

a plurality of access circuits configured to access external network connections; and 

an optional forwarding engine selectable to generate routing information 

corresponding to data received via the access circuits and to provide the routing 

information to the packet buffer. 
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AMENDED CLAIMS 

[received by the International Bureau on 18 September 2002 (18.09.02); 
new claims 2-1 1 added; remaining claims unchanged] 

1 . A network interface controller, comprising: 

a plurality of scatter gather circuits connectable to a host via a bus; 

a packet buffer configured for communication with the scatter gather circuits; 

a plurality of access circuits configured to access external network connections; and 

an optional forwarding engine selectable to generate routing information 
corresponding to data received via the access circuits and to provide the routing 
. information to the packet buffer. 

2. The network interface controller of Claim 1, configured such that: 
the forwarding engine is inactive; 

the packet buffer and access circuits are configured for data communication 
therebetween; and 

the packet buffer and scatter gather engines are configured for data communication 
therebetween; 

whereby the network interface controller is configured to a network interface- 
controller mode of operation. 

3. The network interface controller of Claim 2, wherein: 

the packet buffer and access circuits being configured for data communication 
therebetween includes the packet buffer being configured to utilize paging hardware 
thereof for receiving the data from the access circuits. 

4. The network interface controller of Claim 3, wherein: 

the packet buffer and access circuits being configured for data communication 
therebetween includes the packet buffer being configured to selectively cause the access 
circuits to assert flow control signals to corresponding link partners. 

5. The network interface controller of Claim 2, wherein: 

the packet buffer and access circuits being configured for data communication 
therebetween includes the packet buffer being configured to determine that it has received a 
complete data packet and to transmit the received complete data packet to an appropriate 
access circuit. 

n 

AMENDED SHEET (ARTICLE 19) 



8/27/2007, EAST Version: 2.1.0.14 



WO 02/084499 



PCT/US02/12679 



6. The network interface controller of Claim 1, wherein: 

the scatter gather engines are configured for pass-through data communication 

between the packet buffer and external bus circuitry; 
5 the forwarding engine is configured to route the data communication between the 

access circuits and the packet buffer, 

whereby the network interface controller is configured in a switch mode of 

operation. 

10 7. The network interface controller of Claim 6, wherein: 

the scatter gather engines are configured to: 

perform DMA writes of the pass-through data to a host memory of the 
network interface controller; and 

perform DMA reads of the pass-through data from a host memory of another 
15 network controller. 

8. The network interface controller of Claim 6, wherein: 

the forwarding engine being configured to route the data communication between 
the access circuits and the packet buffer comprises computing a routing vector based on the 
20 data being communicated and providing the routing vector to the packet buffer. 

9. The network interface controller of Claim 8, wherein: 

the packet buffer processes the routing vectors to control the inflow of data to and 
outflow of data from the packet buffer. 

•25 

10. The network interface controller of Claim 1, wherein: 

the forwarding engine is configured to process receive data from the access circuits 
and to generate receive routing information therefrom; and 

the packet buffer is configured to process the receive routing information in order to 
■ 30 selectively provide the receive data to an access circuit or to a scatter gather engine. . 

1 1 . The network interface controller of Claim 1 , wherein: 

12 
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the forwarding engine is configured to process transmit data from the scatter gather 
engines and to generate transmit routing information therefore; and 

the packet buffer is configured to process the transmit routing information and to 
process the transmit data based thereupon. 



13 
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Figure 2 A 
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